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EXECUTIVE SUMMARY 


This final report presents findings from the Reading First Implementation Study: 2008-09. It 
provides information on both the implementation and sustainability of the Reading First Program 
as of late 2008, and it follows two other major studies of the program commissioned by the U.S. 
Department of Education, as required by the Elementary and Secondary Education Act (ESEA) 
of 2001. 


The Reading First (RF) Program was first and foremost designed to improve young students’ 
reading skills, as reading proficiency is critical to students’ success in school. Students must be 
able to read and comprehend text well in order to master subject matter taught in various content 
areas. Disproportionately large numbers of minority and economically disadvantaged children 
struggle with learning to read, and their lack of reading skill in turn results in inadequate 
academic performance. For example, about half of all fourth-grade students eligible for free- or 
reduced-price lunches read at the Below Basic level on the 2009 (and the 2007) National 
Assessment of Educational Progress (NAEP) Reading Assessment,' whereas only 20 percent of 
fourth-graders not eligible for the lunch program read at that level (U.S. Department of 
Education, 2007, 2010). 


The Reading First Program 


The Reading First Program provided resources to states (and from states to selected districts and 
schools) to improve reading instruction. The funding was to be used according to the program 
statute and guidance issued by the U.S. Department of Education (ED). Taken together, the 
guidance and the program’s authorizing statute describe several strategies states and local 
subgrantees were to use to improve students’ reading skills (ESEA, as amended, Title I, Part B, 
Subpart 1, Section 1202 (c)(7)(A)UV)(2); Guidance for the Reading First Program, 2002). 
First, the curricula used in classrooms must reflect scientifically based reading research on the 
essential components of effective primary grade reading instruction: 1) phonemic awareness, 2) 
decoding/phonics, 3) reading fluency, 4) vocabulary, and 5) comprehension. 


Second, states may and local subgrantees must offer comprehensive professional development on 
how teachers should work with academically struggling students, as well as how teachers can 
implement research-based reading instruction. 


Third, the guidance addresses procedures for diagnosis and prevention of early reading 


difficulties through a) using valid, reliable measures to screen students; b) using empirically 
' Performance at the Below Basic level means that fourth-grade students are not performing at the Basic level. 
“Fourth-grade students performing at the Basic level should demonstrate an understanding of the overall 
meaning of what they read. When reading text appropriate for fourth-graders, they should be able to make 
relatively obvious connections between the text and their own experiences and extend the ideas in the text by 
making simple inferences.” (National Center for Education Statistics, retrieved June 18, 2010, from 
http://nces.ed. gov/nationsreportcard/reading/achieveall.asp#grade4). 
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validated intensive interventions to help struggling students; and c) monitoring the progress of 
students experiencing difficulties to ensure that the early interventions are indeed effective. 


In April 2002, ED invited state education agencies to apply for RF grants. State applications 
submitted to ED went through an expert review process that resulted in six-year awards 
dependent on demonstration of progress and congressional appropriations. All states and 
jurisdictions except Puerto Rico were awarded RF grants. Guam and the Northern Mariana 
Islands received grants through consolidated grants to the insular areas that may have been used, 
in part, for Reading First activities. States awarded subgrants to school districts based on a 
competitive process. The program statute required states to give priority to districts and schools 
with the greatest demonstrated need, in terms of student reading proficiency and poverty status. 


The Reading First Program was initially funded at approximately $1 billion per year, with grants 
to states that ranged from $518,781 to $152,898,960 per year, and that supported between 14 and 
905 schools per state, totaling nearly 6,000 RF schools in over 1,800 school districts. In fiscal 
year 2008, the last year of funding for the program, Congress reduced the RF appropriation to 
$393 million, a cut of 61 percent. 


Evaluating Reading First 


The program’s authorizing legislation required ED to conduct an independent evaluation of the 
RF Program. Two major evaluations of Reading First have been completed: the Reading First 
Implementation Evaluation (U.S. Department of Education, 2008 and the Reading First Impact 
Study (Gamse, Jacob, Horst, Boulay and Unlu, 2008), respectively, provided evidence about the 
implementation and the impact of the program. Findings from those two studies indicated that 
reading instruction had changed in ways consistent with key program goals and strategies.” 


The Reading First Implementation Evaluation was based on a nationally representative sample of 
Reading First (n=1,092) and non-RF Title I (n=541) schools, and it focused primarily on aspects 
of implementation at the school and classroom levels by comparing schools that had begun to 
implement RF-funded practices to a set of demographically similar non-funded schools. The 
study purposively included schools that had begun to implement RF at different points in time 
(as well as non-funded schools) to learn more about how long it took schools to implement the 
program. It found that reading instruction in RF schools differed from that in non-RF Title I 
schools in several ways, based on survey responses from principals, reading coaches, and 
classroom teachers in grades K, 1, 2 and 3. RF schools devoted more time to reading instruction 
in K—3 classrooms, and were more likely to: a) have reading coaches who assisted teachers in 
implementing their reading programs; b) use reading materials aligned with scientifically based 
reading research; c) use assessments to guide instruction; d) place struggling readers into 
intervention services; and e) have their teachers participate in reading-related professional 
development. RF schools also varied considerably in their choices of instructional programs and 


* See the Reading First Implementation Evaluation: Final Report and Reading First Impact Study: Final Report, 


both released by the U.S. Department of Education. 


assessments, reflecting the program’s blend of federal guidelines that could be adapted to local 
circumstances. 


The Implementation Evaluation found that RF teachers reported using instructional practices 
emphasized by the Reading First Program. It also found that, over time, teachers in other 
schools increasingly reported using similar practices, and that while significant differences 
reported between the two types of schools persisted, the differences diminished between 2004— 
05 and 2006-07. Reading coaches and teachers in non-RF Title I schools reported increased 
prevalence (over a two-year period) of activities aligned with the principles of Reading First, 
including: a) providing assistance to struggling readers, b) developing teacher knowledge and 
use of materials and strategies aligned with scientifically based reading research, and c) 
facilitating staff participation in professional development in the five components of reading 
instruction emphasized by Reading First (phonemic awareness, phonics, vocabulary, fluency and 
comprehension). The Implementation Evaluation found that RF teachers characterized all five 
components of reading as central to their instruction more frequently than their non-RF Title I 
counterparts. However, the large majority of teachers in both RF and non-RF Title I schools 
consistently reported that scientifically based instruction in all five components of reading 
instruction was central to their teaching. The Implementation Evaluation also examined student 
achievement in RF and non-RF Title I schools, using extant state achievement test scores in third 
and fourth grades. These analyses found statistically significant yet modest evidence that in 
some states, RF schools gained between 2 and 3 percentage points more, on average, from pre- 
RF to post-RF implementation than non-RF Title I schools on the proportion of students who 
met state standards on proficiency tests (in 12 of 24 states in grade 3, and in six of 17 states in 
grade 4). 


The Reading First Impact Study was based on a rigorous quasi-experimental design, regression 
discontinuity (RD), and it included both RF (n=125) and comparison schools (n=123) in 18 sites 
(17 RD and one randomized controlled trial), each of which had equal numbers of RF and 
comparison schools. It was designed primarily to assess the impact of the program on classroom 
instruction and student reading achievement. Findings about instructional practices in reading 
were based on direct observations of reading instruction in grades 1 and 2. The study found 
significant differences in classroom instruction between the RF and comparison classrooms; 
specifically, RF teachers spent significantly more instructional time on the five components of 
reading emphasized by Reading First. Both RF and comparison teachers spent significantly 
more time teaching comprehension and decoding than the other component skills. 


The Impact Study also fielded surveys about instructional practices and supports, using the same 
survey items as used in the Implementation Evaluation. Survey findings indicated evidence of 
positive and statistically significant impacts on multiple practices promoted by the Reading First 
Program, including professional development in scientifically based reading instruction (SBRD, 
support from full-time reading coaches, extended amount of reading instruction, and supports 
available for struggling readers. 
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Finally, the Impact Study examined student reading performance, and found no evidence that 
Reading First had a statistically significant impact on student reading comprehension test scores 
in grades 1, 2, or 3 across the three years of the study. The study found a statistically significant 
and positive impact on first-grade students’ decoding skills in the third year of the study, 
although that finding applied only to one grade and one of three years of data collection. Impacts 
on students’ reading achievement were based upon assessments administered directly by the 
research team. 


The current study, the Reading First Implementation Study: 2008-09, builds on and addresses 
issues with the prior two. The previous studies focused on school-level outcomes and not on 
districts or states, both of which played active roles in the implementation of the program. The 
current study was designed, in part, to address questions about what lessons can be learned from 
the Reading First Program. It includes an examination of ongoing or sustained implementation 
of the program (at the state level), which was a particularly salient issue because, when the 
current study began in September 2007, Congress had begun deliberations about eliminating 
funding for the program. In addition, this study sought to examine patterns of student 
achievement for students who had progressed beyond the third grade, the latest grade served by 
the program; ultimately such analyses were not feasible due to data limitations. 


The current study was designed to answer two broad sets of questions, each focusing on a 
different aspect of sustainability of the program. The first set of questions includes the 
following: What are states’ planned responses to the Reading First budget reduction? and 
Which RF program elements do state-level staff believe can be sustained beyond Reading First? 
These questions address the planned and likely sustainability of the program from the 
perspective of state-level program administrators.* The Reading First Program represented a 
significant federal investment in primary education, and even though funding for the program 
itself has since been eliminated, it may still be of interest to policymakers and educational 
practitioners to learn which specific program elements state directors believed were sustainable. 


The second set of questions includes the following: Does student achievement in later grades 
differ for students exposed to Reading First in K-3 compared with students in Title I Schoolwide 
Program (SWP )* schools? and Is there evidence that patterns of student achievement observed 
for students in RF schools were also observed in students in non-RF Title I SWP. schools in RF 
districts ? These questions address the potential for delayed program effects on students by 
examining patterns of student performance after students have completed the targeted grades. As 
noted above, because of data limitations, the study was unable to address these questions. 


> Reference to states throughout this report also implicitly refers to territories and jurisdictions. 


* Title I Schoolwide Programs operate in schools in which not less that 40 percent of students are from low- 


income families. See ESEA, 2001, Title I, Part A, Subpart 1, Section 1114 (a)(1) 
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In addition, because the Reading First Program encouraged funded states and districts to offer 
services, such as professional development, both to schools designated as Reading First 
subgrantee schools as well as to other (not funded) schools and districts, the question of whether 
aspects of program implementation “spilled over” to all non-funded schools in RF districts has 
always been important. The current study attempted to address spillover and contamination 
questions directly by asking whether the absence of effects observed in earlier evaluations of RF 
schools may have reflected spillover of practices and potentially achievement patterns in 
comparison schools. However, due to data limitations, the current study was unable to report on 
these questions. 


This report therefore addresses the first set of study questions—about states’ planned responses 
to the RF budget reduction and describes state-level staff expectations about which RF program 
elements can be sustained with reduced funding levels—using interviews conducted with state- 
level staff in fall 2008. At that point, some states had already experienced budget reductions and 
could therefore describe actual events, while others had yet to experience budget cuts and could 
therefore only describe anticipated changes. It is important, however, to note that interviews 
were conducted prior to the enactment of the fiscal year 2009 budget and the American Recovery 
and Reinvestment Act of 2009 (ARRA). By significantly expanding Title I funding, ARRA could 
potentially alter resources and funding available to support reading programs in ways not 
anticipated by respondents. 


Summary of Key Findings 
Implementation of the Reading First Program 


The Reading First Program provided funding for 5,880 schools in 1,809 districts across the 54 
states and territories (US Virgin Islands, American Samoa, the Bureau of Indian Affairs, and the 
District of Columbia). The Department of Education awarded Reading First funding to states 
between July 2002 and September 2003, and states subsequently awarded funds to districts. 
Because the Department of Education made awards on a rolling basis, states were at different 
points in their six-year grant cycles when interviews were conducted with state Reading First, 
Title I, and reading or literacy directors. In fall 2008, 50 percent of RF directors reported that 
their states were being affected by the 61 percent budget cuts to the RF Program in school year 
2008-09. The remaining RF directors reported they would not be affected by the budget cuts 
until the 2009-10 school year or later. As a result, responses to interview questions about the 
effects of the budget reductions were answered according to states’ (then) current funding status; 
some states were already experiencing reduced funding, and others responded by hypothesizing 
how the reduction in funding would affect them in the future. 


The Reading First Program operated in a variety of departmental and organizational contexts, yet 
the states’ RF directors all described their job responsibilities similarly. Their responsibilities 
included administering competitions for RF funding, allocating RF funds, providing professional 
development, outreaching to non-RF schools, and monitoring RF evaluation data. For the 
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majority of RF directors, managing the RF program represented their primary responsibility and 
occupied the majority of their professional time. 


At the state level, the RF program experienced substantial changes in leadership over its six-year 
history. For example, only about 17 states (one-third) had had the same leadership over that six- 
year period, and 37 states had had more than two RF directors. At the operational level, 
however, there was remarkable continuity, as 98 percent of districts awarded subgrants at the 
outset of the program were still implementing the RF program in fall 2008, and less than four 
percent of schools had exited the RF program. 


The Reading First Program was designed to improve student reading achievement, specifically 
through reliance on student performance data. Almost all state interview respondents (91 
percent) reported that the RF leadership was the primary state-level user of RF data. Both RF 
state staff and other personnel reported that the overall capacity to use data had improved, 
primarily at the district and school levels. Specifically, principals and teachers in RF schools 
received and used their own school-level data. Additionally, two-thirds of state RF directors 
reported releasing data to school principals of all of their state’s RF schools. Substantial 
proportions of Reading First directors, Title I directors, and states’ reading representatives also 
reported that school capacity to use data had improved (96, 49 and 63 percent, respectively). 


Reading First funds were used to support improved instruction in both RF-funded districts and 
schools as well as in non-funded districts and schools. RF directors reported that offering RF- 
related professional development was the most common form of outreach to non-RF districts and 
schools, and that multiple RF practices were being adopted in non-RF districts and schools, 
including: 


e Professional development (94 percent) 

e Assessment and progress monitoring (63 percent) 
e Materials and curricula (59 percent) 

e Uninterrupted reading blocks (39 percent) 

e Interventions for struggling readers (31 percent) 
e Reading coaches (30 percent). 


Anticipated Changes to Implementation After Budget Cuts 


Over three-quarters (76 percent) of RF directors anticipated a reduction in state technical 
assistance, specifically a reduction in availability of professional development both for RF and 
non-RF districts and schools. However, many also acknowledged that several elements of 
Reading First would be worth sustaining: 


e Use of reading coaches (56 percent) 
e RF materials and curricula (39 percent) 
e Data-driven instruction (35 percent) 
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e Use of assessments (35 percent) 
e Scientifically based reading instruction (33 percent). 


Approximately one-third of RF directors reported that some RF program elements would likely 
be sustained through inclusion in state standards or new literacy or early learning programs; six 
of the 54 states and territories interviewed reported plans to support additional RF cohorts of 
subgrantees. However, two-thirds (67 percent) of states anticipated that they would need 
alternative sources of funding (e.g., Title I, special education) to support their RF-related 
activities. Further, more than one-third (39 percent) of RF directors explicitly stated it was 
unrealistic to sustain RF using other state funds due to financial shortfalls in all aspects of states’ 
budgets. 


Achievement Data Limitations 


The study planned to use a difference in differences approach to estimate the difference between 
the actual achievement of students who had attended RF schools and their predicted 
achievement, which is based on achievement data collected both prior to RF and from students in 
non-RF schools. Data were obtained from 15 states that had collected and maintained individual 
student achievement and enrollment data over a continuous span of time that began at least one 
year before the Reading First Program was implemented in that state. The study sought to use 
data from Title I schoolwide program (SWP) schools to provide a comparison group for the RF 
schools, as the SWP schools served similar populations of students as did RF schools: students 
who were economically disadvantaged and often underperforming. However, students attending 
RF schools were consistently lower-performing, and more economically disadvantaged, on 
average, than students attending Title I SWP schools, since—by definition—REF schools were 
those schools serving the lowest-performing and most disadvantaged students in their respective 
districts. Therefore, it was not possible to construct a comparison group that was sufficiently 
well-matched to the RF study sample to support credible estimates of program impacts, and this 
report does not present any results from student achievement analyses. 


Conclusions 


Fall 2008 interviews with RF directors, Title I directors, and reading representatives across the 
54 states and territories suggested that state staff recognized the immediate need to cut aspects of 
their RF Programs as a result of the 61 percent budget reduction, and that without sufficient 
funding in the future, the program would eventually be dismantled. Yet states also varied in both 
their responses to actual and anticipated federal RF budget cuts and their perceptions about the 
sustainability of specific RF practices. In some states, staff reported efforts to integrate Reading 
First into ongoing reading programs or states’ Response to Intervention programs (RTD); other 
state staff predicted that RF would likely disappear without direct federal funding and leadership. 
Cross-state variation in aspects of implementation and sustainability also reflected differences in 
states’ local contexts. States differed in terms of infrastructure, leadership, resources, and 
multiple other factors that influenced program implementation and likely sustainability. Despite 
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the variation, however, there were consistencies in what states reported about the program as 
implemented and its legacy. 


The study’s findings are timely, as policymakers and legislators are considering strategies and 
policies for both current and future federal investments in literacy (e.g., Title I, Part A). A future 
reexamination of Reading First’s implementation and rollout, as well as its focus, may contribute 
to future programs designed to improve students’ reading and academic achievement, perhaps by 
providing more information on the extent of implementation, including the role of instruction in 
specific components of reading skill. 
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CHAPTER 1: STUDY OVERVIEW 


The Elementary and Secondary Education Act (ESEA) of 2001 (PL 107-110) established the 
Reading First (RF) Program (Title I, Part B, Subpart 1) to address the fact that substantial 
numbers of children were not developing adequate reading skills. The overarching goal of the 
legislation was to ensure that all children could read at or above grade level by the end of third 
grade. With the exception of Title I funding, which supports general reading activities for 
children in low-income schools, the Reading First Program was substantially larger and both 
more ambitious and explicit in its guidance than any previous school-based early literacy 
initiative undertaken in the U.S. 


The legislation required the U.S. Department of Education (ED) to contract with an outside 
entity to conduct an evaluation of the RF Program. ED contracted with Abt Associates Inc. to 
conduct the third national evaluation of the Reading First Program, called the Reading First 
Implementation Study: 2008-09.” This study began in fall 2007 and built upon an earlier 
implementation study. 


The Reading First Program 


Reading First was designed to improve young students’ reading skills, as reading proficiency is 
critical to students’ success in school.° Students must be able to read and comprehend text well 
in order to master subject matter taught in various content areas. Disproportionately large 
numbers of minority and economically disadvantaged children struggle with learning to read, 
and their lack of reading skill in turn results in inadequate academic performance. For example, 
about half of all fourth-grade students eligible for free- or reduced-price lunches read at the 
Below Basic level on the 2009 (and the 2007) National Assessment of Educational Progress 
(NAEP) Reading Assessment’ whereas only about 20 percent of fourth-graders not eligible for 
the lunch program read at that level (U.S. Department of Education, 2010; 2007). 


The Reading First Program was intended to promote instructional practices that prior research 
had found effective (Elementary and Secondary Education Act, 2001). Taken together, the 
guidance and the program’s authorizing statute describe several strategies states and local 
subgrantees were to use to improve students’ reading skills (ESEA, as amended, Title I, Part B, 


° The other two evaluations include the Reading First Implementation Evaluation and the Reading First Impact 


Study, both conducted by Abt Associates for the U.S. Department of Education. 


Note that Title I, Part A, supports reading instruction and Title I funding could therefore be used to support 
teaching essential components of reading instruction. 


Performance at the Below Basic level means that fourth-grade students are not performing at the Basic level. 
“Fourth-grade students performing at the Basic level should demonstrate an understanding of the overall 
meaning of what they read. When reading text appropriate for fourth-graders, they should be able to make 
relatively obvious connections between the text and their own experiences and extend the ideas in the text by 
making simple inferences.” (National Center for Education Statistics, retrieved May 10, 2006, from 
http://nces.ed. gov/nationsreportcard/reading/achieveall.asp#grade4). 
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Subpart 1, Section 1202 (c)(7)(A)(V)(2); Guidance for the Reading First Program, 2002). 
First, the curricula used in classrooms must reflect scientifically based reading research on the 
essential components of effective primary grade reading instruction: 1) phonemic awareness, 2) 
phonics, 3) reading fluency, 4) vocabulary, and 5) comprehension. 


Second, the states may and local subgrantees must offer comprehensive teacher professional 
development on the implementation of scientifically based reading practices; states were required 
to offer comprehensive professional development on how teachers should work with 
academically struggling students as well as how teachers could implement research-based 
reading instruction. 


Third, the guidance addressed procedures for diagnosis and prevention of early reading 
difficulties through a) using valid, reliable measures to screen students; b) using empirically 
validated intensive interventions to help struggling students; and c) monitoring the progress of 
students experiencing difficulties to ensure that the early interventions are indeed effective. 


In April 2002, ED invited state education agencies to apply for RF grants. State applications 
submitted to ED went through an expert review process that resulted in six-year awards 
dependent on demonstration of progress and congressional appropriations. All states and 
jurisdictions except Puerto Rico were awarded RF grants.® Guam and the Northern Mariana 
Islands received grants through consolidated grants to the insular areas that may have been used, 
in part, for Reading First activities. States then awarded subgrants to school districts based on a 
competitive process. The program statute required states to give priority to districts and schools 
with the greatest demonstrated need, in terms of student reading proficiency and poverty status. 
The program began funding states (and in turn districts and schools) as of July 2002, and 
program implementation began immediately. 


The Reading First Program provided detailed guidance to states (and through states to districts 
and schools) about implementing the activities described above. However, states could and did 
select different instructional materials, assessments, and types and providers of professional 
development. States were able to heed the federal Reading First Guidance and simultaneously 
make decisions based on local priorities. 


Initially, the Reading First Program was funded at approximately $1 billion per year, with grants 
to states that ranged from $518,781 to $152,898,960 per year, and that supported between 14 and 
905 schools per state, totaling nearly 6,000 RF schools in approximately 1,900 school districts. 
In Fiscal 2008, the last year of funding for the program, however, Congress reduced the annual 
RF budget to $393 million, a cut of 61 percent. 


* Guam and Northern Mariana Islands received grants through consolidated grants to the insular areas. 
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Evaluations of Reading First 


Findings from two major evaluation reports on Reading First provided evidence that reading 
instruction had changed in ways consistent with key program goals and strategies, although the 
more rigorously designed Reading First Impact Study found no evidence of statistically 
significant impacts on students’ reading comprehension, and the Reading First Implementation 
Evaluation found weak evidence of improved reading achievement but with a considerably 
weaker design.” In particular, the Reading First Implementation Evaluation found that reading 
instruction in RF schools differed from that in non-RF Title I schools in several ways, based on 
survey responses from principals, reading coaches, and classroom teachers in grades K, 1, 2 and 
3. RF schools devoted more time to reading instruction in K—3 classrooms and were more likely 
to: a) have reading coaches who assisted teachers in implementing their reading programs; b) use 
reading materials aligned with scientifically based reading research; c) use assessments to guide 
instruction; d) place struggling readers into intervention services; and e) have their teachers 
participate in reading-related professional development. RF schools also varied considerably in 
their choices of instructional programs and assessments, reflecting the program’s dual emphasis 
on federal guidelines and local flexibility. 


The Reading First Implementation Evaluation was based on a nationally representative sample of 
Reading First (n=1,092) and non-RF Title I (n=541) schools, and it focused primarily on aspects 
of implementation at the school and classroom levels by comparing schools that had begun to 
implement RF-funded practices to a set of demographically similar non-funded schools. The 
study purposively included schools that had begun to implement RF at different points in time 
(as well as non-funded schools) to learn more about how long it took schools to implement the 
program. It found that reading instruction in RF schools differed from that in non-RF Title I 
schools in several ways, based on survey responses from principals, reading coaches, and 
classroom teachers in grades K, 1, 2 and 3. RF schools devoted more time to reading instruction 
in K—3 classrooms and were more likely to: a) have reading coaches who assisted teachers in 
implementing their reading programs; b) use reading materials aligned with scientifically based 
reading research; c) use assessments to guide instruction; d) place struggling readers into 
intervention services; and e) have their teachers participate in reading-related professional 
development. RF schools also varied considerably in their choices of instructional programs and 
assessments, reflecting the program’s blend of federal guidelines that could be adapted to local 
circumstances. 


The Implementation Evaluation found that RF teachers reported using instructional practices 
emphasized by the Reading First Program. It also found that, over time, teachers in other 
schools increasingly reported using similar practices, and that while significant differences 
reported between the two types of schools persisted, the differences diminished between 2004— 
05 and 2006-07. Reading coaches and teachers in non-RF Title I schools reported increased 
prevalence (over a two-year period) of activities aligned with the principles of Reading First, 


° The two reports are Reading First Implementation Evaluation Final Report and Reading First Impact Study: 


Final Report, both released by the U.S. Department of Education. 
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including: a) providing assistance to struggling readers, b) developing teacher knowledge and 
use of materials and strategies aligned with scientifically based reading research, and c) 
facilitating staff participation in professional development in the five components of reading 
instruction emphasized by Reading First (phonemic awareness, phonics, vocabulary, fluency and 
comprehension). The Implementation Evaluation found that RF teachers characterized all five 
components of reading as central to their instruction more frequently than their non-RF Title I 
counterparts. However, the large majority of teachers in both RF and non-RF Title I schools 
consistently reported that scientifically based instruction in all five components of reading 
instruction was central to their teaching. The Implementation Evaluation also examined student 
achievement in RF and non-RF Title I schools, using extant state achievement test scores in third 
and fourth grades. These analyses found statistically significant yet modest evidence that in 
some states, RF schools gained between 2 and 3 percentage points more, on average, from pre- 
RF to post-RF implementation than non-RF Title I schools on the proportion of students who 
met state standards on proficiency tests (in 12 of 24 states in grade 3, and in six of 17 states in 
grade 4). 


The Reading First Impact Study was based on a rigorous quasi-experimental design, regression 
discontinuity (RD), and it included both RF (n=125) and comparison schools (n=123) in 18 sites 
(17 RD and one randomized controlled trial), each of which had equal numbers of RF and 
comparison schools. It was designed primarily to assess the impact of the program on classroom 
instruction and student reading achievement. Findings about instructional practices in reading 
were based on direct observations of reading instruction in grades | and 2, and findings about 
student achievement were based upon direct assessments of students by the research team. 


The study found significant differences in classroom instruction between the RF and comparison 
classrooms; specifically, RF teachers spent significantly more instructional time on the five 
components of reading emphasized by Reading First. Both RF and comparison teachers spent 
significantly more time teaching comprehension and decoding than the other component skills. 
The Impact Study also fielded surveys, using the same survey items as used in the 
Implementation Evaluation. Survey findings indicated evidence of positive and statistically 
significant impacts on multiple practices promoted by the Reading First Program, including 
professional development in scientifically based reading instruction (SBRI), support from full- 
time reading coaches, extended amount of reading instruction, and supports available for 
struggling readers. 


The evidence about impacts on classroom instruction did not translate into observed differences 
in students’ reading comprehension. The Impact Study found no evidence that Reading First had 
a Statistically significant impact on student reading comprehension test scores in grades 1, 2, or 3 
across the three years of the study. The study found a statistically significant and positive impact 
on first grade students’ decoding skills in the third year of the study, although that finding 
applied only to one grade and one of three years of data collection. 


In September 2007, ED’s Policy and Program Studies Service initiated the current evaluation, 
the Reading First Implementation Study: 2008—2009. The evaluation began before Congress 
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reduced the program’s budget and then eliminated funding altogether. Given that the program 
no longer exists, one could ask whether an evaluation of Reading First is relevant and potentially 
useful for educational policy and early reading research. While the Reading First Program itself 
has been eliminated, it is likely that there will be other federal literacy initiatives (e.g., the 
Literacy Education for All, Results for the Nation legislation'®), or that reauthorization of the 
Elementary and Secondary Education Act will include an explicit focus on literacy. There may 
be policy lessons derived from both the implementation and observed impacts of the Reading 
First Program on students’ reading comprehension that could apply to both current and future 
federal investments in literacy (e.g., Title I, Part A), particularly for programs that combine well- 
articulated federal guidance with local interpretation. There may also be lessons about the 
content of the Reading First Program, which was based on a strong foundation of evidence about 
early reading instruction, yet the findings from prior research had not been tested as a coherent 
program before Reading First was launched on a national scale. The current study was designed 
to address questions about what lessons can be learned about sustainability from the Reading 
First Program. The study’s design features are summarized below. 


Study Design 


The study was initially designed to answer two sets of broad questions, each focusing on a 
different aspect of sustainability. The first set of questions included the following: What are 
states’ planned responses to the Reading First budget reduction? and Which RF program 
elements do state-level staff believe can be sustained beyond Reading First? These questions 
address the planned and likely sustainability of the program from the perspective of state-level 
program administrators. The Reading First Program represented a significant federal investment 
in primary education, and even though funding for the program itself has since been eliminated, 
it may still be of interest to policymakers and educational practitioners may be able to learn 
which specific program elements were deemed sustainable from state directors. 


The second set of questions included the following: Does student achievement in later grades 
differ for students exposed to Reading First in K-3 compared with students in Title I Schoolwide 
Program (SWP)'' schools? and Is there evidence that patterns of student achievement observed 
for students in RF schools were also observed in non-RF Title I SWP schools in RF districts ? 
These questions address the delay of program effects by examining patterns of student 
performance after students have completed the targeted grades. 


The study sought to use data from Title I SWP schools to provide a comparison group. Such 
schools served similar populations of students as did RF schools: students who were 
economically disadvantaged and often underperforming. However, students attending RF 
schools were consistently lower-performing, and more economically disadvantaged, on average, 


See summary of proposed LEARN legislation at: http://www.govtrack.us/congress/bill.xpd?bill=s111- 
2740&tab=summary. 


Title I Schoolwide Programs operate in schools in which not less that 40 percent of students are from low- 
income families. See ESEA, 2001, Title I, Part A, Subpart 1, Section 1114 (a)(1) 


Chapter 1: Study Overview 5 


than students attending Title I SWP schools, since—by definition—RF schools were those 
schools serving the lowest-performing and most disadvantaged students in their respective 
districts. It was therefore not possible to construct a comparison group that was sufficiently 
well-matched to the RF study sample to support credible estimates of program impacts. 


State Interviews 

This final report addresses questions based on interviews with state RF directors, Title I 
directors, and other state-level personnel responsible for statewide reading decisions in the fall of 
2008 (from all states and territories, including the District of Columbia, Guam, American Samoa, 
U.S. Virgin Islands and the Bureau of Indian Affairs), that received RF grants. In the fall of 
2008, the study team interviewed key state staff in all states, including the state RF director, as 
well as the state Title I director and a state reading representative. As described above, by the 
fall of 2008, the RF Program had already experienced substantial reductions in funding, and the 
program seemed likely to face elimination. With that context in mind, the interviews provided 
information about key district and state policies that affected reading instruction in both RF and 
comparison schools, as well as information about states’ responses to changes in federal support 
for Reading First. The interviews included questions on the following topics: 


e Management and leadership of Reading First 
e RF funding at the state level 


e Relationship and coordination of Reading First with other state-level initiatives and 
programs 

e Reading First Data on Student Reading Performance 

e Outreach to non-Reading First districts and schools 

e Variation and divergence across districts between RF and non-RF schools’ reading 
instruction 


e Long term sustainability of Reading First 


Interviews were completed with the state personnel responsible for overseeing the Reading First 
programs in their respective states, the 54 RF state-level directors, as well as other individuals 
with state-level responsibility for elementary reading programs and oversight of the state-level 
Title I program (47 State Title I directors and 32 state reading, language arts, or literacy 
directors, respectively). Data from state interviews were analyzed to address the topics 
summarized above, using qualitative techniques as well as some simple descriptive analyses. 


The Organization of This Report 
The report includes two chapters. Chapter | reviews the context for the study, the RF legislation, 


and the study design. Chapter 2 addresses the study’s evaluation questions about sustainability 
using findings from interviews with state Reading First and other personnel. 
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CHAPTER 2: STATE RESPONSES TO READING FIRST BUDGET 
REDUCTION AND SUSTAINABLE PROGRAM ELEMENTS 


This chapter addresses the first set of evaluation questions: What were states’ planned responses 
to the Reading First budget reduction?, and Which RF program elements did staff believe could 
be sustained beyond Reading First funding? To answer these questions, the study conducted 
interviews with key state staff in all states, including the state Reading First directors, state 

Title I directors, and state reading or literacy directors (e.g., state reading, language arts, or 
literacy directors who had oversight of early grades’ reading instruction). Note that references to 
states throughout this report also subsume references to territories and jurisdictions, and Guam 
and the Northern Mariana Islands received grants through consolidated grants to the insular areas 
that may have been used, in part, for Reading First activities. The interviews provided 
information about key district and state policies that affected reading instruction in both RF and 
comparison schools, as well as information about states’ responses to reductions in federal 
funding for Reading First. 


Key Findings 
e There was considerable variation in RF program operations, in the planned program changes as a 
result of reduced federal funding, and predictions about which elements of RF programs would likely 
be sustained. 
e 50 percent of RF directors were affected by federal budget cuts in fiscal year 2008-09; others were 
anticipating substantial changes in 2009-10 or later. 
e 76 percent of RF directors anticipated reduction in state TA (specifically in availability of professional 
development for RF and non-RF districts and schools). 
e 67 percent of RF directors anticipated that their states would seek alternative federal funds to support 
RF-related activities (e.g., Title |, soecial education funds). 
e Nearly all of RF directors (91 percent) indicated that they themselves and other RF leaders were 
primary users of RF evaluation data. 
e State staff reported improved capacity to use data at district and school levels. 
e RF directors reported that the primary form of outreach to non-RF schools and districts was via RF- 
related professional development. 
e RF directors reported that a variety of teaching strategies being implemented with RF funds were 
likely to be sustained, funds permitting: 
o Use of reading coaches (56 percent) 
o RF materials and curricula (39 percent) 
o Data-driven instruction (35 percent) 
o Use of assessments (35 percent) 
o Scientifically based reading instruction (83 percent) 
e One-third of RF directors reported that sustaining RF elements could occur through state standards 
or new literacy or early learning programs. 
e More than one-third (39 percent) of RF directors said it was unrealistic to expect that RF would be 
sustained using other state funds. 
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Interviews were conducted in the fall of 2008. At the time, all state staff knew that federal 
funding for the program was sharply curtailed; however, because the program began funding 
states at different points between July 2002 and September 2003, states were at different points 
in their six-year grant cycles. Half of the states were already (in fall 2008) operating their 
programs with reduced federal RF budgets; the other states were expecting reductions to take 
effect in the next school year (or even later).'? Before describing the data collection and analysis 
in more detail, the text below reviews key features of the Reading First Program that may have 
influenced states’ experiences. 


The Reading First Program provided funding to states to carry out scientifically based reading 
programs that reflected both common guidance and local flexibility. The Department of 
Education issued guidance to states (which were then translated into guidelines for districts and 
schools) about how resources could be used. States could allocate funds flexibly, by allocating 
resources up to statutory maximums rather than on a strictly formulaic basis to different 
categories of spending (e.g., subgrants to districts, materials, assessments, and professional 
development). Specifically, states were required to use at least 80 percent of their award funds to 
provide subgrants to local school districts, and states could reserve up to 20 percent of their 
Reading First funds to support staff development, technical assistance to non-funded districts and 
schools, and planning, administration and reporting. Further, once the Department of Education 
approved their grant applications, states were able to make their own decisions about choices 
within given categories (e.g., which materials, reading programs, assessments, and professional 
development providers made sense in their respective local contexts). As a result, states varied 
considerably in their RF activities, programs, and resources. 


Data Collection and Analysis 


The following chapter outlines the approach to answering the first set of evaluation questions: 
What were states’ planned responses to the Reading First budget reduction and Which RF 
program elements did staff believe could be sustained beyond Reading First funding ? 


To answer these questions, the study conducted interviews with key state staff in all states, 
including the state Reading First directors, state Title I directors, and state reading or literacy 
directors (see Appendix 1 for a copy of the interview protocol). The interviews provided 
information about key district and state policies that affected reading instruction in both RF and 
comparison schools, as well as information about states’ responses to reductions in federal 
funding for Reading First. The study team summarized the data gathered from these interviews 
using a combination of qualitative and quantitative analytic techniques (see Appendix 2). 


There are limitations inherent to using qualitative interviewing. The precision of the information 
obtained through interviews depends on the willingness and ability of respondents to provide 
complete and accurate responses. Given the seriousness of the budgetary implications for the 


'2 States may also have begun to experience budget issues unrelated to reductions in Reading First funding, as the 


economy nationally experienced substantial contraction in the fall of 2008. 
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Reading First directors—including the possibility that their positions could be eliminated—they 
likely faced more pressure than Title I or reading or literacy directors, and their responses could 
well have reflected biases related to program continuity. The study completed interviews with 
all of the Reading First directors, and only a subset of Title I and reading or literacy directors. 
Another limitation was the considerable variation in the Reading First directors’ experience and 
program knowledge. Some Reading First directors had held their positions for nearly six years; 
others for less than one year, resulting in differences in the depth and quality of their knowledge. 
The information obtained about state and district policies and responses to reductions in funding 
for the Reading First Program may have been affected by respondent experience and knowledge. 
Finally, due to the differences between states in the timing of RF awards, some were already 
implementing their Reading First Programs under the 61 percent budget reduction while others 
were anticipating the reductions. As a result, responses to questions about the effects of funding 
cuts were answered quite differently by those who could comment about actual operations than 
those who were predicting what their states would do at some point in the future. 


Measures 


State Profile Summaries 

Prior to interviewing state staff, the study team prepared state profile summaries to organize the 
most recent data available in fall 2008 (when interviews were conducted) on states’ RF 
implementation. These data were gathered from a variety of sources: state websites, a database 
about Reading First that was maintained by the Southeast Regional Educational Laboratory 
(SEDL), previously submitted Annual Performance Reports, and other publically available 
sources of information. State websites typically had information about resources available for 
teachers and schools, upcoming (and in some instances, recently provided) professional 
development and technical assistance, assessments, and reports or results. Some state websites 
also had lists of the funded districts and schools. The SEDL website provided information about 
district award dates (last updated in June 2007). The Annual Performance Report data, 
accessible through ED, summarized information on the specific assessments each state used to 
report performance for its Reading First schools. 


This pre interview data review helped prevent the collection of duplicative information from 
state staff, and allowed the study team to use interview time to efficiently verify and update the 
information already on hand. The study team created state-specific summary profile worksheets 
that included information on the amount and timing of RF funding, number of districts and 
schools in each state, number of cohorts awarded, required assessments, resources available for 
RF and non-RF districts, and available reports and evaluation data (Appendix 3). 


State Interview Protocol 
The state interview protocols collected information through semi-structured telephone 


interviews, using a combination of scripted open-ended and follow-up prompts. 


As described in more detail below, the interview questions asked about such topics as state-level 
management and leadership of Reading First, Reading First funding, the relationship and 
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coordination of Reading First with other state-level reading initiatives and programs, uses of 
Reading First data, outreach to non-Reading First districts and schools, and the long-term 
sustainability of Reading First. The state interviews intentionally included more Reading First— 
specific questions for the states’ RF directors. 


The state interview protocol covered the following topics: 


e Management and leadership of Reading First. Background and responsibilities of the 
state coordinator. 


e RF funding at the state level. How many rounds of funding states awarded to districts 
or schools; whether (and if so why) any districts and schools withdrew from participation 
in RF before their funding ran out; planning for future RF funds and cohorts; staff 
stability or turnover. 


e Relationship and coordination of Reading First with other state-level initiatives and 
programs. Whether and how Reading First was coordinated or integrated with other 
state-level reading-related initiatives. 


e Reading First Data on Student Reading Performance. How the state prepared its RF 
data and to whom the state made it available; who used such data; decisions that were 
influenced by these data about professional development, technical assistance, district 
and school performance, future funding; observed changes and variation in state, district, 
and school capacity to use data. 


e Outreach to non—Reading First districts and schools. Structure of outreach to non-RF 
districts and schools; topics of interest to non-RF districts and schools. 


e Variation and divergence across districts between RF and non-RF schools’ reading 
instruction. Similarities or differences within the state’s RF districts; full scale district 
implementation of Reading First across all four grades at once or staggered 
implementation; level of flexibility for RF district-level decision-making about reading 
instruction (similarities or differences in reading instruction, requirements for a reading 
block, support from a coach, etc., at RF and non-RF schools.) 


e Long term sustainability of Reading First. Aspects of the RF program that were likely 
to persist in the wake of reduced (or eliminated) RF funding. 


Administration of State Reading First Programs 


The following section summarizes findings on the administration of states’ Reading First 
programs, focusing on the time period prior to the budget cuts for the 54 states and jurisdictions 
that had been awarded RF grants. Data are based on telephone interviews with state RF 
directors, Title I directors, and reading representatives (state-level personnel responsible for 
statewide reading decisions). 
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Characteristics of State Reading First Program Directors and Education Agencies 


Length of Service 

The U.S. Department of Education began approving state-level Reading First Program 
applications in 2002. All applications, with the exception of one, were approved by the end of 
2003. Thus, 53 out of 54 RF programs had been active for five to six years in fall 2008. In 16 
states (30 percent), Reading First directors had held that position for six or more years (Exhibit 
1). RF directors in 12 states (23 percent) reported that they had been in that position for one year 
or less. Reading First directors in the remaining states, approximately half, reported being in 
their positions for two to five years. RF directors with less experience also have had less 
historical or institutional knowledge of their respective state RF program than those with more 
experience. This difference may have been reflected in their responses to interview questions 
that ask about programmatic changes over the course of the five- to six-year period. 


Exhibit 1: Number of Years as Reading First Director, 2008 


1 year orless 


6 or more years 23% 
30% 


2-5 years 
47% 


EXHIBIT READS: Twenty-three percent of the 53 responding Reading First directors reported having served as RF 
director for one year or less. 


Source: Interviews with RF and other state personnel. 


Reading First Directors’ Responsibilities and Time Spent on the Reading First Program 


Nearly two-thirds (63 percent) of state RF directors reported spending 90 percent or more of 
their time on Reading First. Their responsibilities typically included administering competitions 
for RF funding; developing budget allocations; conducting professional development; providing 
outreach to non-RF schools; and monitoring RF evaluation data. Their specific roles ranged 
from direct involvement at the school or classroom level to oversight of external consultants or 
internal RF program staff whose primary responsibility was for a particular aspect of the 
program. Those RF directors whose portfolios included multiple program areas (in addition to 
Reading First) reported having responsibility for allied program areas, such as early literacy, 
Title I, and other literacy-related programs. 
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Competitions for RF Funding: Most RF directors (80 percent) reported having been 
involved in the competition for the RF funding process in some capacity, as a state 
employee. Those who had not been involved had assumed their role as state RF director 
after their respective states’ grant competition had been completed. 


Budgets and Funding Allocations: In determining budgets and funding allocations, RF 
directors’ responsibilities ranged from oversight of a grant manager or administrator 
responsible for budgeting and funding allocation to annual review and approval of state 
and school district RF budgets. 


Professional Development Activities: Involvement of RF directors in professional 
development varied more widely than for other tasks. Reported roles included planning 
professional development activities at the state or local level; providing oversight of 
professional development providers (external contractors and/or internal RF staff); 
designing and leading professional development activities with state reading staff, school 
leaders, reading coaches or classroom teachers; training professional development 
providers as part of a “train the trainers” model; coordinating professional development 
efforts with other state offices (particularly Special Education and English Learner (EL) 
offices); and communicating professional development offerings to districts and schools. 


Outreach to non—Reading First Districts: The primary outreach activity reported by 
almost all RF directors (96 percent) was to extend invitations to professional 
development activities to non-RF personnel. The majority of RF directors actively 
recruited at non-RF schools, inviting personnel to attend RF professional development 
offerings. Others provided oversight to ensure that RF districts conducted outreach by 
making presentations or distributing newsletters. 


Monitoring Reading First Evaluation Data: Reading First directors’ monitoring 
responsibilities included oversight of staff responsible for monitoring evaluation data; 
frequent, direct monitoring of data to plan professional development activities; 
completion of Annual Performance Reports; monitoring progress of individual schools; 
provision of technical assistance to districts and schools in how to use and interpret 
assessment results; and coordination with external evaluators. Almost half of the RF 
directors explicitly reported working with an external evaluator on monitoring evaluation 
data. 


State Organizational Structure and Coordination Between Offices 


The organization of state offices can have significant implications for interoffice communication 
and collaboration. Interviews with state RF directors, Title I directors and reading 
representatives revealed substantial variation in the organization and coordination between the 
various state offices: 
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RF offices were housed in a variety of locations within state education agencies; some 
were co-located in joint offices with Title I or with language or other federal grant 
programs, while others were stand-alone units. 
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e Coordination between Reading First and other programs or offices was not a universally 
evident practice. Some states reported frequent and long-standing collaborative 
relationships between programs or offices; others reported a more “siloed” environment. 


Number and Nature of Reading First Awards 


Reading First directors confirmed data on the RF awards in their states from the Southwest 
Educational Developmental Laboratory (SEDL) database and the ReadingFirstDataOnline 
website. The RF program provided funding for approximately 5,800 schools in 1,850 districts 
across the 54 states and territories, (American Samoa, the Bureau of Indian Affairs, the District 
of Columbia, and the U.S. Virgin Islands) since its inception. 


The number of RF schools ranged from 14 schools to 905 schools across states and territories 

(Exhibit 2 and Exhibit 3), representing between 3 and 23 percent of the primary grade schools. 
States varied both in terms of the number of schools funded and the proportion those numbers 
represented. 


A number of factors determined how state and local awards were made. First, the RF legislation 
required that the amount of the RF award must equal the school’s Title I, Part A, distribution 
from the preceding fiscal year. This amount was formulaic and non-negotiable. Second, the 
school-level award had to be of sufficient size and scope to enable schools to improve reading 
instruction through adherence to the principal components of the RF program. This amount 
varied from state to state and involved factors such as student enrollment, the number of K—3 
students reading below grade level, the number of children with special education needs, and the 
number of children for whom English is not their first language. Costs related to the hiring of 
reading coaches, teacher professional development, the purchase of core and intervention reading 
materials as well as materials for diagnoses, monitoring and assessment also were considered. 
State award amounts ranged from $1,279,429 to $137,397,187 in the program’s first year, and 
from $9,113,091 to nearly $1 billion ($930,706,299) over the course of the program’s subsequent 
years. 


Approximately one-third of states received grant awards for a full six years; the majority of 


states awarded local grants for two or more years. Only two states (4 percent) awarded single- 
year grants. All but two states implemented their RF programs in all four grades (K—3) at once. 
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Exhibit 2: Numbers of Primary and Reading First Schools, by State, 2005-06 School Year 


Number of IN TUTtn) ole) mre) Mm at-r-(ollare| Percent of Primary 
Primary Schools First Schools Schools Reading First 
Alabama 725 92 8% 
Alaska 169 14 12% 
American Samoa 23 23 1% 
Arizona 1,116 156 7% 
Arkansas 585 99 6% 
California 5,676 905 6% 
Bureau of Indian Affairs schools 93 27 3% 
Colorado 991 81 12% 
Connecticut 558 25 22% 
Delaware 108 14 8% 
District of Columbia 133 37 4% 
Florida 1,961 584 3% 
Georgia 1,280 141 9% 
Hawaii 182 69 3% 
Idaho 346 30 12% 
Illinois 2,564 381 7% 
Indiana 1,154 71 16% 
lowa 780 108 7% 
Kansas 780 Mb 19% 
Kentucky 783 73 11% 
Louisiana 714 113 6% 
Maine 404 25 16% 
Maryland 876 39 22% 
Massachusetts 1,170 88 138% 
Michigan 2,164 166 13% 
Minnesota 1,048 51 21% 
Mississippi 449 66 T% 
Missouri 1,239 114 11% 
Montana 432 33 138% 
Nebraska 755 29 26% 
Nevada 336 27 12% 
New Hampshire 299 15 20% 
New Jersey 1,539 75 21% 
New Mexico 472 88 5% 
New York 2,516 255 10% 
North Carolina 1,358 98 14% 
North Dakota 287 25 11% 
Ohio 2,151 113 19% 
Oklahoma 967 73 138% 
Oregon 660 50 13% 
Pennsylvania 1,905 159 12% 
Rhode Island 209 16 138% 
South Carolina 633 49 138% 
South Dakota 349 21 17% 
Tennessee 986 74 138% 
Texas 4,103 705 6% 
US Virgin Islands 24 15 2% 
Utah 520 23 23% 
Vermont 256 16 16% 
Virginia 1,182 88 13% 
Washington 1,212 86 14% 
West Virginia 468 42 11% 
Wisconsin 1,236 56 22% 
Wyoming 201 17 12% 


EXHIBIT READS: Alabama has 725 primary schools and 92 RF schools; 13 percent of Alabama’s primary schools are RF. 
Source: http://www.sedl.org/readingfirst/report-title].html#chart (downloaded Oct. 3, 2007), and NCES Table Builder, at 
http://nces.ed.gov/ccd/bat/output.asp (School year 2005—06, # Schools by School level code and State) (downloaded Oct. 3, 
2007) 
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Exhibit 3: Number of RF Schools, by State, 2008 


AK 


1 Oo, 


H! (D> 


Number of Number 
Participating (Percent) 


Schools of States 
14-30 15 (28%) 
31-75 17 (31%) 


76-125 13 (24%) 
> 125 9 (17%) 


EXHIBIT READS: Fifteen states (28 percent) had funded between 14 and 30 RF schools. These states included Alaska, 
Delaware, New Hampshire, US Virgin Islands, Rhode Island, Vermont, Wyoming, Connecticut, South Dakota, Tennessee, 
American Samoa, Utah, Maine, Nebraska and Nevada. 


Note: While not shown pictorially, the number of funded schools was between 14 and 30 for US Virgin Islands and American 
Samoa, and between 31 and 75 for the Bureau of Indian Affairs. 


Source: Interviews with RF and other state personnel. 


Unanticipated Exits From the Program 


Exiting the RF program was defined as a cessation in program participation, either voluntarily or 
involuntarily. There were very few unanticipated district or school exits from the RF program. 
According to RF directors, less than 2 percent of RF-funded districts prematurely exited the 
program. 


The majority of states (42) reported that some small number of districts or schools exited 
Reading First prematurely. Most of these exits were voluntary. Funding was revoked for only 
18 districts and 55 schools. Generally, funding was revoked for two reasons: 1) poor 
implementation of the RF program, and 2) inadequate gains in reading achievement relative to 
what RF schools were expected to achieve. 
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Several state RF directors pointed out that Reading First was not an easy program to implement 
because it was relatively prescribed, requiring specific elements (e.g., a 90-minute uninterrupted 
reading block). They additionally noted that the program foundation emphasized building 
teachers’ knowledge in multiple and complex domains, including: 


1. Understanding of scientifically based reading research about the five components of 
reading; 

2. Use of knowledge of scientifically based reading to evaluate and best utilize reading 
curricula; and 


3. Understanding the diagnosis and prevention of early reading difficulties through a 
range of assessment and monitoring tools. 


State, district or school-level personnel needed to ensure that professional development (often 
through RF coaches) was offered to teachers so that knowledge of these three domains was 
sufficient to carry out the RF program. In addition to the required domains, the state RF 
directors noted the requirement for more student testing and related analysis, as well as 
interpretation and reporting of test results. 


In total, 42 states reported some involuntary exits; for 26 of these, unanticipated exits from the 
RF program were so small (e.g., one or two schools), they did not have an impact on subsequent 
funding decisions. Thirteen states, however, redistributed money to already participating RF 
districts or schools. Three states (with a total of two districts and six additional individual 
schools making unanticipated exits) used the money to fund more schools. 


Thirty-four states reported a total voluntary withdrawal of seven districts and 86 additional 
individual schools from the RF program before the end of their funding cycle, predominantly due 
to school closings. In addition, eight RF directors reported that four districts and 37 schools did 
not apply for continuation funding. 


Reading First Student Achievement Data 


Assessment of students’ reading proficiency was a key requirement of the Reading First 
Program. The RF legislation required schools to use valid and reliable assessments to screen, 
diagnose students’ reading difficulties, and to measure student reading outcomes (PL. 107-110, 
Title I, Part B, Subpart 1). Eligible school districts that received RF subgrants were required to 
use funds to carry out a number of assessment-related activities including “[s]electing and 
administering screening, diagnostic, and classroom-based instructional reading assessments” 
(Section 1202.c.7.A.1.). 


Additionally, teachers in grades K—3 in RF schools as well as special education teachers of 


kindergarten through grade 12 were allowed to receive professional development that included 
“instruction in the use of screening, diagnostic, and classroom-based instructional reading 
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assessments and other procedures that effectively identify students who may be at risk for 
reading failure or who are having difficulty reading” (Section 1202.c.7.A.iv.ILbb). 


School districts were required to collect and summarize data “to document the effectiveness of 
activities carried out . . .in individual schools and in the local educational agency as a whole; and 
to simulate and accelerate improvement by identifying the schools that produce significant gains 
in reading achievement” (Section 1202.c.7.A.v.I and II), as well as report data for all students 
and categories of students (described in Section 1111(b)(2)(C)(v) dD). 


Release, Distribution, and Use of Student Performance Data 


At the state level, the RF student achievement data largely remained under the purview of the 
state RF office. Over half of RF directors (59 percent) reported that the RF student achievement 
data were shared with their state RF leadership team to support their work with schools and 
districts, and about one-third of directors (33 percent) reported making the data available to their 
state superintendent’s or commissioner’s office. One-fifth of directors (20 percent) reported 
making the data more widely available to their state departments of education, and only four 
states (7 percent) also reported sharing their RF student achievement data with their state 
legislature to help inform statewide reading initiatives and policy. 


According to RF directors, student achievement data were used primarily at the school level. 
Almost three-fourths of state RF directors (72 percent) reported that the school-based coaches 
used the data with the teachers whom they coached. In all states, principals and teachers in RF 
schools received their own school-level data, and two-thirds of state RF directors also reported 
releasing the data to building principals of all of their state’s RF schools. 


The RF student achievement data were sometimes made available to the public; approximately 
one-fifth of RF directors (19 percent) reported making their state’s RF achievement data public. 
Additionally, 9 percent of state RF directors specifically noted releasing the RF student 
achievement data to the media. 


Interviewees reported that RF achievement data were reviewed or used by a variety of audiences, 
from RF state leadership only (“No one else really uses the data”’) to a distribution list that 
included, for example, the governor, the state board of education, task forces on reading or 
literacy, state offices of assessment, Special Education or exceptional learners, departments of 
English language learners, the School Improvement division, and the office of curriculum and 
instruction. 


Ninety-three percent of RF directors reported that their RF leadership team was the primary 
state-level user of RF data. This users’ group was typically comprised of the state director, 
professional development directors, data coordinators, and technical assistance providers, all 
within the RF state office. 
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More than three-fourths of RF directors (78 percent) reported that the Reading First achievement 
data were used by the state RF leadership to plan professional development and technical 
assistance for RF schools. In addition, more than half of the state RF directors (54 percent) 
reported using the Reading First student achievement data to measure school and district 
performance. Forty-three percent reported using the achievement data to determine future or 
continued funding of Reading First, and a similar proportion (41 percent) reported using these 
data to compare RF and non-RF schools. 


The majority of state RF directors (89 percent) also reported that they expected districts and 
schools to use data to inform local decision-making and planning. Expected actions included 
creating local action plans, making changes in professional development and instituting other 
local policy changes. 


Example Responses: Release, Distribution, and Use of Student Performance Data 


“With the available data at the school level, they are expected to have the principal and coach meet with teachers to decide how 
to tweak the program to meet the needs of the students. They include that data in their plan each year when they apply for funds 
from other sources, Title V, and in their triangulation data. At the district level, they use the school reports and include them in 
their annual consolidated application. At the school level, they plan their professional development from it and target specific 
teachers.” (Reading First director) 


“Districts and schools are absolutely expected to make changes in professional development and instruction as a result of the 
data. The state works closely with regional technical assistance centers in helping districts and schools write action plans for 
improvement. At the school level, the schools are expected to disaggregate their results down to the grade level and have 
conversations about those results and develop grade-level plans for improvement.” (Reading First director) 


“The school day has recently been extended for all teachers, in part to allow time for teacher collaboratives. These 
collaboratives provide time for the teachers to get together and take a close look at data to see where students need help. The 
teachers want to see changes in their classrooms. Another tremendous change is the movement away from whole group 
instruction and toward differentiated instruction. By using the data, teachers know how to group students for effective small 
group instruction. By doing this, teachers can meet individual student needs instead of teaching to the middle.” (reading 
representative) 


State and Local Capacity to Use Reading First Data 


State Capacity 

RF directors were asked to describe their perceptions of state and local staff capacity to use RF 
data. Two-thirds of state RF directors reported that they had observed increased state capacity to 
use RF data since the program’s inception. Respondents also described Reading First as having 
provided data-based linkages between state, district, and school levels; collaboration between the 
state leadership, district leadership, and school personnel was strengthened when all could focus 
on the same readily-accessible student achievement data. Directors discussed what appeared to 
be an increased state sophistication in data manipulation and tools such as Web-based systems. 
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Example Response: State and Local Capacity to Use Reading First Data 


“At a state level, and | don’t say this to be politically correct, we have always done a good job at looking at data. Our office has 
now done a better job of bring data to districts to make comparisons from where they are and where they want to be. These 
conversations are now tied to instruction. We have been able to help districts to have conversations about the achievement gap. 
Using the data has helped us to plan for Professional Development. We use the data to determine who needs further intervention 
through Professional Development.” (Reading First director) 


The majority of RF directors (67 percent) attributed the changes in their state’s capacity to use 
data explicitly to the RF Program. The RF legislative requirements to use progress monitoring, 
diagnostic, and benchmark data often provided the states with their first opportunity to follow 
frequently collected student data so closely. 


A few respondents noted that RF data efforts were being used by state assessment or 
accountability offices, as well as School Improvement and Special Education divisions or 
bureaus. In these instances, the student data required by the RF initiative were used by other state 
level initiatives to measure progress toward School Improvement goals and the effectiveness of 
Special Education reading interventions. A handful of respondents linked the use of data to drive 
instruction in Reading First to the newer Response to Intervention (RTI) initiative. As part of the 
Individuals with Disabilities Education Act (IDEA) 2004 legislation, RTI is a comprehensive, 
multistep process that closely monitors how students are responding to different types of services 
and instruction. RTI provides an improved process of pre-referral interventions. For example, 
one respondent stated “RF is the rudder of the whole RTI model. Everything that came out of RF 
isin RTI. They are almost a mirror image of each other.” 


Interestingly, several Title I directors attributed the increased capacity in data use to the broader 
implementation of ESEA, and several reported the alignment and coordination of RF data efforts 
with similar and/or larger data efforts throughout the state, e.g., the state superintendent’s focus 

on data, or an earlier statewide data initiative. 


District and School Capacity 


Reading First directors, Title I directors and reading representatives reported that districts and 
schools appeared to have become more sophisticated consumers of data since the start of RF. 
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Example Responses: District and School Capacity to Use RF Data 


“There has been a tremendous difference, particularly at the school level. At first, there was a lot of resistance to the process. It 
took about a year to 18 months for the schools to realize what a great resource they had in all of this data being available. | see 
a huge change now: schools put up assessment walls to monitor progress, have collaboration meetings that include Title | and 
Special Education teachers in the discussions of the data, have worked to use the data to develop specific interventions as 
needed, and are constantly updating the tiers. Now reading instruction is really data-driven in our schools.” (Reading First 
director) 


“In the past five or six years we've seen a pretty dramatic improvement in staff ability to use data to guide decision-making, PD 
and instruction. The decisions made are based on data. Staff have a better ability to interpret and understand the data.” (Title | 
director) 


“Using data has changed in so many ways. Principals also use it in terms of planning workshops, and it is down to the indicator 
level. They know exactly not just reading comprehension but specifically what aspects of reading comprehension teachers need 
help with. Their use of data is a lot more powerful than anything they did before.” (reading representative) 


Reading First Program in Fall 2008 


The U.S. Department of Education (ED) began funding states in July 2002 with six-year grants. 
Because ED approved states’ applications over a 15-month period (June 2002 through September 
2003) and states’ internal RF funding processes varied in length, the end date of states’ RF grants 
ranged from fiscal year 2008 to fiscal year 2009 and beyond. About half of RF directors 
interviewed in fall 2008 reported their Reading First programs already had been directly affected 
by the federal budget cuts. Interestingly, the magnitude of reduced funding varied across those 
states because some states had offset the 61 percent federal reduction with carryover monies (i.e., 
RF funding that had been allocated for, but not used, during the prior school year) and other 
funding sources. Respondents who reported that budget cuts were not yet directly affecting their 
state’s RF programs anticipated reduced funding during the 2009-10 school year or later. These 
latter states generally used carryover funding from prior years or had begun RF school-level 
implementation in the 2004—05 school year. 


How states chose to operationalize reduced RF funding also varied. Some states reported 
reducing program elements (either by reducing or eliminating professional development or 
technical assistance provided by the state’s RF office); other states also changed district or 

school award amounts as well. The differences in the timing of budget cuts and how states 
operationalized budget reductions clearly influenced respondents’ experiences and perceptions of 
likely sustainability of Reading First. 


The following sections describe how states implemented four elements of the Reading First 
Program prior to reductions in funding (technical assistance, outreach to non-RF schools, 
adoption of RF practices in non-RF districts and schools, and collaboration among RF program 
offices and other state-level initiatives). Each section also includes a discussion of how the RF 
element was or would be affected as a result of funding reductions. Note that because some 
states had already experienced reduced budgets in fall 2008, the discussion of the effects of the 
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reductions incorporates both actual and anticipated reductions in funding. The section concludes 
with a discussion of states’ responses to reduced federal funds for Reading First. 


Technical Assistance 


All 54 RF directors reported that their states provided some type of technical assistance to 
districts and schools, and all but three of the 54 states described providing technical assistance in 
multiple forms. The most common forms of technical assistance reported by states include 
Professional Development (PD) offerings (91 percent), school visits (69 percent) and convening 
of regularly scheduled group meetings (37 percent). The three most common forms of technical 
assistance are described below. 


Professional Development. Ninety-one percent of states reported providing PD as an element of 
technical assistance. Professional development included reading academies, summer institutes, 
coach institutes, leadership institutes, and other conferences, institutes, or workshops. 
Respondents reported that these professional development offerings were intended to address the 
following topics and content areas: 


e Administering Dynamic Indicators of Basic Early Literacy Skills (DIBELS) or other 
assessments; 


e Analyzing data and making instructional decisions; 
e Teaching the five essential components of reading instruction; 
e Implementing a core reading program; and 


e Developing coaching skills; 
— Implementing Language Essentials for Teachers of Reading and Spelling (LETRS) 
training; 
— Planning interventions; and 
— Targeting instruction. 


Reading First directors generally characterized professional development as activities that 
occurred once or twice a year over multiple days (e.g., Summer Institutes) as more typical, 
whereas some respondents described PD offerings as occurring at various times throughout the 
year. The professional development offerings were designed primarily for classroom teachers 
and reading coaches, although some respondents mentioned that school principals were 
encouraged to attend or described additional offerings developed exclusively for school 
administrators. 


RF directors reported that the following groups were direct providers of professional 
development: 


e State RF staff, including the Reading First director; 


e University partners; 
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e National reading experts; 
e RF Technical Assistance Centers; 
e Publishing companies; and 


e Other organizations involved in providing materials or assessments. 


Approximately one-quarter (24 percent) of states who reported offering professional 
development described offering courses as an element of their PD. Unlike institutes and 
workshops that required in-person attendance, courses were available in multiple formats, 
including in-person attendance and Internet-based applications or modules. The content of 
courses, while varied, was reported to be centered on RF principles. Only a few states reported 
that their RF courses counted toward graduate credits or endorsements on teaching credentials or 
licenses. 


School Visits. Over two-thirds of RF directors (69 percent) reported that they themselves 
conducted school visits. The purpose of state-conducted school visits, as reported by RF 
directors, was to provide direct technical assistance to schools by giving feedback on classroom 
observations, discussing how to analyze and use data to make instructional decisions, attending 
grade-level meetings, holding meetings with the principal, and addressing any issues or questions 
from the school community. Some respondents reported that school visits helped them monitor 
implementation and that struggling schools would generally receive greater support. School 
visits were reported to involve administrators, reading coaches, and classroom teachers. The 
visits most often were conducted by state regional RF coordinators, sometimes called RF 
specialists, each of whom were assigned a number of schools in their geographic region. A few 
respondents indicated that these regional coordinators were associated with universities. RF 
directors in smaller states (both geographically and in terms of number of RF schools) were more 
likely to describe conducting school visits personally, in conjunction with regional coordinators 
or other RF staff. 


Group Meetings. Thirty-seven percent of RF directors reported their states held regular group 
meetings and described them as occasions for convening specific groups of individuals to work 
together on areas of common interest. For example, reading coaches were brought together to 
model lessons for each other and discuss strategies to implement what they learned in trainings. 
Groups of school administrators discussed sustainability, examined budgets, looked at data, or 
learned about leading the Reading First initiative within the school. Some group meetings were 
designed to bring together all levels of staff involved in Reading First, including district staff, 
principals, reading coaches, and classroom teachers. Topics for these broader group meetings 
included discussing data and conducting book studies. Such meetings generally occurred 
quarterly, with some of the larger meetings only occurring annually or bi-annually. Group 
meetings primarily were facilitated by state RF staff, including the RF director and regional 
specialists. 
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Changes in Technical Assistance Resulting from Budget Cuts 

While approximately half of RF directors had already had to adjust activities to align with budget 
reductions, more than three-quarters (76 percent) of RF directors expected the Reading First 
budget reduction to limit the overall technical assistance and RF professional development their 
states would be able to provide. At that time, cost had been the primary reason for reduction 
across all types of technical assistance. With the anticipated budget reductions, respondents 
cited that RF staffing levels could not be maintained, as the expenses of hiring external 
consultants or travel costs for providers or participants to attend trainings or conduct visits could 
not be sustained. A few states reported that they would maintain some aspects of their 
professional development offerings because they believed they were important and valuable 
opportunities for their teachers, either by making changes in their providers (e.g., from experts or 
consultants to state RF staff or experienced reading coaches), or continuing professional 
development with support from another state-level reading initiative. 


Some RF directors specifically noted that the budget cuts would have a negative impact on 
professional development in the form of course work (both a reduction in development of new 
course work offerings and availability of existing course work); others reported that online 
courses and modules that had already been developed would continue to be a component in 
sustaining professional development activities. RF directors described alternative means of 
reaching out to and facilitating group meetings, using online meetings, newsletters, or email. 
Although a few states reported that they would maintain the frequency of school visits, others 
reported plans to maintain frequent contact with schools via email, phone, or videoconferencing. 
Respondents indicated that while such use of technology had cost-saving benefits over travel, 
they generally did not consider it an adequate substitute for on-site support through visits. 


Outreach to Non-RF Districts and Schools 


The majority (83 percent) of state RF directors reported inviting non-RF personnel to participate 
in professional development trainings that were provided in-person and hosted by the state or 
districts or that was offered as course work made accessible to all online. States appeared to be 
following the Reading First legislation that allowed staff in non-RF schools to participate in RF- 
sponsored professional development. Most states did indicate professional development 
opportunities were open to all staff. 


Forty-one percent of states described outreach as a district-level responsibility or effort, while the 
remaining states monitored district outreach efforts from the state office. One method of 
outreach described by state RF directors included implementing RF elements across all schools 
in RF-funded districts; alternatively, non-RF schools within districts that housed RF schools 
extended opportunities for staff in non-RF schools to visit and observe RF schools or receive 
some assistance from RF reading coaches. State Reading First directors commented that it may 
have been less likely that RF practices were being implemented in districts with no RF schools, 
especially in the states where outreach was primarily a district-level responsibility or effort. 
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Alternative means of outreach discussed by state RF directors included the following: 


School visits to help implement RF practices more broadly (five states); 


Presentations (in-person, radio programs, and press releases) by state RF staff (five 
states); 


Websites with RF information (e.g., about scientifically based reading research and 
curricular materials such as lesson plans, planning maps, literary schedules) (five states); 


Providing materials to non-RF schools (three states); and 


Disseminating regular newsletters to provide information about Reading First, reading 
instruction, and upcoming events (two states). 


State RF directors were asked to describe their approaches to RF outreach. The following three 
methods were most frequently mentioned: 
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Approximately one-quarter of RF directors discussed spreading RF principles to as many 
schools as possible, prioritizing outreach to non-RF schools from the outset; 


Over half of RF directors indicated they focused initially on implementing RF practices 
in RF-funded schools and gradually spread RF practices to non-RF schools, once the RF 
schools were up and running; and 


Less than one-quarter of RF directors said they focused their attention primarily on 
implementing RF practices in RF-funded schools and where possible/cost-effective 
opened up RF-related opportunities to non-RF schools. 


States encountered challenges to providing outreach to non-RF districts and schools; 
approximately three-fourths (74 percent) of RF directors reported challenges to outreach. The 
most frequently cited challenges were: 
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Inability to meet demand due to lack of capacity, funds, or space at professional 
development offerings; 


Travel distance; 


Lack of understanding by non-RF school staff that they were eligible to attend Reading 
First events; 


Additional demands on schools’ time; 
Coordinating with other offices; and 


Disparities in knowledge between RF and non-RF school staff (e.g., knowledge gaps that 
required RF staff to tailor training to non-RF schools, the inability to gather as much or 
the same type of data from non-RF schools). 
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Despite these challenges to outreach, one-third (33 percent) of RF directors explicitly mentioned 
that non-RF districts and schools made requests to the state Reading First office for outreach. 
About half of those respondents described requests for specific professional development or 
training, and the other half described requests from non-RF districts and schools to learn more 
about the elements of the RF program, including: choosing a core program, implementing 
assessments, structuring a reading block, conducting classroom observations of reading 
instruction, and improving reading teaching strategies. 


Changes in Outreach to Non-RF Districts and Schools Resulting from Budget Cuts 
Respondents anticipated the budget cuts would have similar effects on both RF and non-RF 
districts and schools. Because RF-sponsored professional development was available to all 
schools, all staff would therefore receive less professional development. Four respondents noted 
that non-RF districts and schools would likely experience significant reductions in resources 
after budget cuts, because limited resources might serve to limit participation to only RF district 
and school staff or result in states instituting fee-for-service policies for non-RF participants. 
Similarly, another four respondents mentioned that previously available stipends, scholarships, 
or professional development travel grants for non-RF districts and schools would likely be 
reduced or cut entirely after budget reductions. 


Adoption of Reading First Practices in Non-Reading First Districts and Schools 


States reported making efforts to provide opportunities for non-RF districts and schools to 
participate in Reading First activities. State RF directors identified a number of key 
programmatic elements of RF when describing spillover of RF principles and practices to non- 
RF schools (See Exhibit 4). It is important to note that when state personnel reported that certain 
practices had been adopted by their states’ non-RF schools, they did not necessarily define how 
common the practices were in their non-RF schools (i.e., what percentage of non-RF schools had 
adopted these practices) as they were focusing their attention on RF implementation in the 
schools receiving RF funding. Therefore, it is important for the reader to keep in mind that 
according to state RF directors, the following practices reportedly had spread to some, but not 
necessarily all, of the non-RF schools in the state. Whenever available, data on the percentage of 
non-RF schools in the state involved in adoption of RF principles and practices are reported. 
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Exhibit 4: RF Elements RF Directors Report Were Adopted by Non-RF Schools, 2008 


0% 25% 50% 75% 100% 

Professional Development 
Assessments/ Progress Monitoring 
Materials & Curncula 


Uninterrupted Reading Block 


Interventions for Struggling 
Readers 


Reading Coaches 


EXHIBIT READS: Eighty-one percent of states reported that professional development was a practice being adopted by 
non-RF districts and schools. 

Note: Data were available for 54 states. 

Source: Interviews with RF and other state personnel. 


Professional Development. Over four-fifths (83 percent) of RF directors mentioned professional 
development when describing their states’ outreach activities; similarly, 81 percent of RF 
directors referenced professional development again when describing spillover. 


Assessments and Progress Monitoring. Almost two-thirds (63 percent) of states reported that 
non-RF schools used RF assessments. Nearly half of these respondents (44 percent) also 
reported that these non-RF schools used data from assessments to determine instruction. 


Materials and Curricula. Over half (59 percent) of states reported that RF materials and 
curricula were adopted in non-RF districts and schools in their states. Respondents commonly 
mentioned core reading programs; a handful also indicated that non-RF schools in their state 
utilized supplementary or intervention materials related to RF. A few respondents noted that 
non-RF schools used “scientifically based” reading programs. 


Uninterrupted Reading Block. More than one-third (39 percent) of states reported that 
uninterrupted reading blocks were implemented by non-RF schools in their state. Respondents 
indicated that the use of uninterrupted reading blocks was fairly widespread across non-RF 
schools, and that the practice was “fairly popular” or found “in all schools.” Some respondents 
mentioned that a dedicated reading block was already mandated as part of state legislation or as 
part of a larger reading initiative with similar principles as RF. 


Interventions for Struggling Readers. Nearly a third of respondents (31 percent) reported that 
non-RF schools in their states implemented special interventions for struggling readers. These 
responses described differentiated or tiered instruction as part of the interventions to support 
struggling readers in non-RF schools. 


Reading Coaches. About one-third of respondents (30 percent) reported that non-RF schools in 
their state used reading coaches. Respondents reported that the use of reading coaches was 
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facilitated at the district-level, saying, for example, “Lots of districts have done a really good job 
of reallocating funds to support an internal facilitator (AKA coach) position because they see it’s 
working in RF schools.”’ Respondents noted that district funding, Title I funds, or other funds 
were often sources for fiscal support of reading coaches in non-RF schools. 


Changes in Adoption of Reading First Practices in Non-Reading First Districts and Schools 
Resulting from Funding Reductions 

Many Reading First directors, in particular, observed that the support for and spread of RF-like 
practices in non-RF schools would decrease once budget cuts were implemented because states 
would have to reduce the amount of professional development provided. As one respondent 
explained, this concern was exacerbated by turnover of teachers who had RF-related training and 
the influx of teachers who had not had the training: 


Example Response: Anticipated Changes to Adoption of RF Practices in Non-RF District and Schools 


“Once the program is curtailed, the positive effects of the program—of having this focus on reading instruction in Reading First 
and non-Reading First schools—could also be curtailed. With the loss of money, the biggest impact will be in the professional 
development for teachers. If the professional development goes away, new teachers won't have exposure to the same heavy 
focus on reading instruction. We might all lose some of that focus. The efficacy of individual teachers in conveying that 
knowledge (the components of reading instruction) to students will be impacted.” (Title | director) 


Conversely, other respondents expressed a slightly more positive view of their state’s capacity to 
sustain the implementation of RF principles and practices in non-RF schools, noting that when 
people find something that works, they will find a way to sustain it despite lack of funding. Yet 
others indicated that implementation of some RF-like practices does not require funds, inferring 
that sustainability need not always be directly linked to funding availability. Nevertheless, while 
non-RF schools could, to some extent, address sustainability at the school-level, respondents 
indicated that state and district support for adoption of their RF practices was an important 
component of sustainability in non-RF schools, and that such support would likely decrease 
under a reduced (or nonexistent) budget. 


Some states also indicated that spillover would continue in cases where the state reading 
initiative(s) or frameworks were founded in or shared RF elements. 
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Collaboration Among RF and Other State-Level Initiatives 


This section describes collaboration and integration between Reading First and other state-level 
offices and programs, as well as anticipated changes in collaboration between RE and other 
programs given the reduction in federal funding for Reading First. In this section, responses are 
reported at the state level.’* 


Collaborations between state RF programs and other state-level initiatives were commonplace 
across the 54 states and territories. All RF directors reported that their RF offices collaborated 
with at least one other program and/or initiative at the state level. Some initiatives were 
frequently named as collaborative partners. Exhibit 5 presents the percentage of states that 
indicated collaborations with various state-level initiatives or programs. 


Exhibit 5: Percent of RF Directors Reporting Collaborations Between RF and Other State 
Programs, 2008 
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EXHIBIT READS: Thirty-five states (65 percent) reported collaborating with the Special Education program(s) in their 

state. 

Note: Data were available for 54 states. 

* Higher education generally meant one or more university partners rather than a higher education department or office within 
the state department of education 

* Title II refers to programs funded through Title II of the Elementary and Secondary Education Act, Title II supports the 
preparation, training and recruiting of high quality teachers and principals. 

* Title II refers to programs funded through Title III of the Elementary and Secondary Education Act, which addresses 
programs for language instruction for limited English proficient and immigrant students. 

Source: Interviews with RF and other state personnel. 


While interviews were conducted with the Reading First director, Title I director, and literacy leader in each 
state, the data in this section are reported at the state level. In order to move from the multiple responses given 
within a state to a state-level summary, responses from the interviewees within each state were combined. A 
response was included in the state-level response if it was given by at least one respondent in the state. 


It is important to note, however, that about one-fifth of the 47 Title I directors and 32 reading representatives 
interviewed individually responded “I don’t know” to all or most of the questions about collaborative partners. 
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Some RF programs’ collaborations were driven by the RF legislation. For example, the 
legislation required that Special Education (SPED) teachers attend RF professional development, 
and SPED collaborations often involved providing professional development for SPED teachers. 
Additionally, a number of Higher Education partners assisted states in providing the professional 
development and technical assistance required by the RF legislation. 


Example Responses: Collaboration among EDs and Other State-Level Initiatives 


“[RF] is working together with resource teachers in providing PD to the SPED folks. SPED folks help the classroom teachers 
with reading or they have their own classrooms in reading ... [RF] also worked with SPED to tailor assessments to the SPED 
population. They are working with SPED on progress monitoring, that is, to have the students tracked day by day, week by 
week, year by year.” (reading representative) 


“The Reading First and Title | Directors meet every week to make sure we are aligning and leveraging each other's funds and 
programs and discussed how to use Title | funds to support Reading First work once Reading First funds end.” (Title | director) 


While all states reported that their RF program had collaborative partners in other state-level 
initiatives and programs, the nature, intensity, and focus of these collaborations varied notably 
from partner to partner and state to state. Some respondents reported that their collaborations 
consisted primarily of periodic meetings; other respondents noted that they saw or emailed their 
collaborative partners on a daily basis to ensure constant sharing of information. Additionally, 
some collaborations were described as having been continuous since the inception of RF, while 
others had only been in place for a few months. It is important to note that some respondents 
characterized their collaborations as joint endeavors between RF and other initiatives, whereas 
other respondents described collaborations in terms of keeping their partners informed of their 
program developments. There were also pragmatic and compliance-related aspects of some 
collaborations, including, for example, working with Title I to ensure that federal funds were 
used appropriately. 


Challenges to Collaboration. Respondents discussed challenges encountered in endeavoring to 
or collaborating with other state-level initiatives and programs. In most states (81 percent), 
respondents described having encountered one or more challenges in forming collaborations. 
The most frequently mentioned challenges include: inability to collaborate with a specific entity; 
lack of time; and “silo” mentality in some departments. 
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Example Responses: Challenges to Collaboration 


Inability to collaborate with a specific entity: 

“_..[WJorking with Higher Ed has really been the most challenging. Philosophically, they’re still in the Whole Language and 
Balanced Literacy Era; they've done their life’s work on this. To switch over [to SBRR] would be very contradictory to what 
they've done professionally.” (reading representative) 


“RF hasn't gotten much positive press at the national level, so people hesitate to come to the table.” (Reading First director) 


Lack of time: 
“Time. It is not that people aren't willing or open, it’s just that in every given day there are so many hours. And everyone has lots 
of their own pieces to take care of.” (Title | director) 


Prevailing “silo” mentality: 
“At the beginning | believe there was an attempt to collaborate with other offices, but with many bureaucracies, there is ’siloing’ of 


responsibilities, heavy focus on compliance, and low staff.” (Title | director) 


Changes in Collaborations Resulting from Budget Cuts 

About two-thirds of respondents commented specifically about the changes they anticipated in 
RF-related collaborations as a result of the 61 percent reduction of federal funding for Reading 
First. Most of these respondents noted that collaborations would likely decrease with reduced 
funding. Interestingly, however, a small group of respondents predicted that collaboration would 
actually increase because partners would have to take on more RF-related responsibilities. 


States’ Response to Reduced Federal Funds for Reading First 


The reduction in federal funding led RF directors to explore multiple measures to support the 
currently funded districts and schools. Respondents described efforts to seek alternative funding 
sources, strategies and challenges to sustaining Reading First and plans for future cohorts. 


Alternative Funding Sources 

Two-thirds (67 percent) of RF directors reported that their states planned to find alternative 
federal or state funding sources to help support RF districts and schools. Exhibit 6 presents some 
of the cited alternative funding sources. 


It is important to note that while the majority of states were in the process of implementing or 
planning strategies to help sustain RF in late 2008 by using one or more of the aforementioned 
funding sources, more than one-quarter of RF directors explicitly said that their respective states 
did not have plans to find other funds to sustain RF. 


30 Chapter 2: State Responses to Reading First Budget Reduction and Sustainable Program Elements 


Exhibit 6: RF Directors’ Planned Alternative Funding Sources for Reading First Schools and 
Districts, 2008 
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EXHIBIT READS: Fifty-six percent of states reported that they planned to utilize Title I funds as an alternative funding 
source for Reading First schools and districts in their state. 


Note: Data were available for 53 of 54 states. 


* Title V refers to programs funded through the Elementary and Secondary Education Act promoting informed parental choice 
and innovative programs. 


Source: Interviews with RF and other state personnel. 


Strategies to Support RF 

A variety of strategies were being employed or developed by states to sustain Reading First 
principles once funding was eliminated. States described using one or more of the following 
approaches to sustainability under a reduced budget: 


e Focus limited resources on fewer schools; 
e Spread resources to as many schools as possible; and/or 


e Primarily fund school-level RF program elements, while reducing or eliminating state- 
level spending on professional development and technical assistance. 


Less than one-fifth of states reported planning to focus their funding on school-level aspects of 
the program, reducing their technical support and provision of professional development from 
the state RF office. One of these states reported: 


Example Response: Focus Funding on School-level Elements of RF 


“... The year-to-year level of schools’ grant amount is not reduced from what the first cohort of schools received. The real 
drawback...is not in the funding level, it is that the state-provided professional development, technical assistance, and perhaps 
evaluation are really having trouble with funding, and these schools will probably not get the same level of assistance as the first 
cohort of schools.” (Reading First director) 
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Other strategies mentioned by states included: 


e Hosting sustainability workshops or trainings to educate districts and schools about 
methods they could employ to help ensure continuation of RF program and practices 
(e.g., having outside experts share sustainability approaches; discussing best practices for 
ensuring sustainability; learning about pilot studies from schools whose funding has 
ended; and meeting individually with schools to discuss their unique situations); 


e Testifying at the state legislature on behalf of RF to share the program’s successes and 
ask for earmark funding for continuation of RF; 


e Planning to incorporate elements of RF into future expenditures; 


e Using technology to replace face-to-face RF trainings and communications in order to 
reach more participants using fewer resources; and 


e Incorporating the elements of RF into a new program that would replicate RF policies 
and practices statewide. 


Sustainable Elements of Reading First 


Reading First directors were asked about methods by which their states were sustaining elements 
of Reading First, and to describe which of those elements were most sustainable given budget 
reductions. Approximately one-third of RF directors (30 percent) explicitly referenced the 
inclusion of RF program elements into state standards or new literacy or early learning programs 
as a method by which Reading First would be sustained. The elements of Reading First that state 
RF directors reported could be sustained after the budget cuts were implemented are grouped 
into ten elements below (Exhibit De 


Reading First directors discussed numerous elements of the RF program they planned to 
continue implementing post-funding. Over one-third of directors discussed sustaining the use of 
reading coaches, RF materials and curricula, data-driven instruction, use of assessments, and 
scientifically based reading instruction. Those RF elements that appeared less likely to be 
sustained included data teams and reading instruction leadership teams. RF directors noted, 
however, that the sustainability of any of these elements was not guaranteed due to the role 
districts and schools play in ensuring that RF elements would continue to be used by schools. In 
discussing the most commonly cited sustainable practice, the use of reading coaches in schools, 
respondents emphasized that this was often a district-level decision, and given cost concerns with 
maintaining non-teaching staff, they expected this practice to be scaled back or eliminated. 
Additionally, 22 percent of respondents commented that the uninterrupted reading blocks might 
become shorter. 


'* Category percentages were calculated based on respondents’ report of one or more elements within a category. 
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Exhibit 7: Elements of RF Reported by RF Directors as Sustainable, 2008 
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EXHIBIT READS: Thirty states (56 percent) reported that they believed the use of Reading Coaches was a sustainable 
element of Reading First. 


Note: Data were available for 53 of 54 states 
Source: Interviews with RF and other state personnel. 


Summary 


Interviews with state staff also revealed considerable variation in (then) current RF program 
operations, as well as in the planned program changes as a result of reduced federal funding. 
The RF directors offered different predictions about which elements of their RF programs would 
likely be sustained. Reading First directors from 27 states and territories (50 percent) reported 
that their states already had experienced the effects of the 61 percent cut in fiscal year 2008-09. 
Reading First directors from the remaining 22 states for whom data were available (41 percent) 
reported they would not be affected by the budget cuts until fiscal year 2009-10 or later. 


The interviews also illustrated that RF directors had experienced challenges in their efforts to 
implement the RF program. Almost three-quarters (74 percent) of RF directors reported having 
encountered impediments to outreach, including inability to meet the demand from non-RF 
districts and schools, lack of awareness among non-RF districts and schools about the 
availability of Reading First professional development, additional demands on their time, 
difficulty in coordinating between offices, and knowledge disparities between RF and non-RF 
staff. Over three-quarters of RF directors (81 percent) also reported challenges to collaboration 
with other programs or initiatives such as differences in philosophy for collaboration, lack of 
time to collaborate, and a “silo” mentality across various state offices. 


All Reading First directors reported providing technical assistance to their RF program schools. 
Almost all (91 percent) reported providing professional development as an element of technical 


Chapter 2: State Responses to Reading First Budget Reduction and Sustainable Program Elements 33 


assistance, 69 percent reported conducting school visits, and 37 percent reported convening 
regular group meetings. Most Reading First directors (83 percent) reported offering RF 
professional development as the most common form of outreach to non-RF districts and schools. 
Less commonly cited forms of outreach included school visits, presentations, newsletters, and 
other RF-related materials. States also reported that a variety of RF practices were being adopted 
by non-RF districts and schools, including use of: professional development (94 percent), 
assessment and progress monitoring (63 percent), materials and curricula (59 percent), 
uninterrupted reading block (39 percent), interventions for struggling readers (31 percent), and 
reading coaches (30 percent). 


State staff anticipated changes to their RF programs as a result of the federal budget cuts for 
Reading First. Over three-quarters of RF directors (76 percent) anticipated that the reduction in 
federal funds would result in less state technical assistance. Reading First directors also 
anticipated that the federal budget cuts would affect the availability of professional development 
availability for Reading First and, to a greater degree, non-RF districts and schools. States also 
anticipated that the existing collaborations between RF and other state programs or initiatives 
would decrease. Sixty-seven percent of states reported efforts to seek alternative sources of 
funding for their RF programs (e.g., Title I and special education monies) to address the federal 
reduction in funds for Reading First. However, more than one-third of states (39 percent) 
explicitly noted that sustaining Reading First with other state funds was unrealistic, because 
states were experiencing budget shortfalls in all areas. 


The elements of Reading First that respondents believed would most likely to be sustained were 
the use of reading coaches, RF materials and curricula, data-driven instruction, use of 
assessments and scientifically based reading instruction. Additionally, although only six states 
reported plans for additional RF cohorts, approximately one-third of RF directors mentioned that 
RF program elements would likely be included in state standards or new literacy or early 
learning programs. 
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APPENDIX 1: READING FIRST COORDINATOR, TITLE | STATE 
DIRECTOR, READING OR LITERACY COORDINATOR 
TELEPHONE INTERVIEW PROTOCOL 


The text below represents instructions to researchers: 

Each Reading First coordinator, Title I state director, and reading/literacy director will have 
received a cover letter describing the topics for the interview. Each interviewer will have 
thoroughly reviewed available data (e.g., state websites, summaries of prior interviews from first 
Implementation Study, information from SEDL and/or Program Office on number of RF districts 
and schools) prior to each interview. 


The questions are numbered below. The bulleted items represent sub-topics or prompts to use if 
necessary. Questions with an asterisk will only be asked of Reading First Directors. 


The text below represents the script that was read by researchers to interview respondents: 


o Name: 
o State: 
o  Position/Title: 


Abt Associates and its partner Hezel Associates are conducting a national study of the 
implementation and sustainability of the Reading First (RF) program. As part of this study, we 
are collecting data from Reading First, Title I, and Elementary Reading/Curriculum Directors to 
learn about the ongoing implementation and sustainability of the Reading First Program in your 
state. State personnel will be interviewed once in the fall of 2008. 


This conversation will include questions about management and monitoring of RF, funding 
decisions, especially given recent federal budget cuts to the Reading First Program, technical 
assistance, how Reading First interacts with other reading initiatives in your state, use of student 
reading achievement data, and outreach to non-RF schools and districts. 


We will be taking notes during our conversation, and may use a tape-recorder to ensure 
accuracy. If you do not wish this conversation to be taped, let us know and we will not use one. 
Your responses will be used to create a state interview summary, and as a result, we cannot 
assure confidentiality; however, all information from your state (and other states) will be 
presented only in the aggregate, and no individuals or states will be identified by name. If you 
have any further questions that we may not be able to answer about this evaluation or this 
conversation, please contact Marianne Beauregard, the director of Abt Associates’ Institutional 
Review Board, at (617) 349-2852, or Beth Gamse, the Abt project director of this study at 617- 
349-2808. Please note these are toll calls. 


According to the Paperwork Reduction Act of 1995, no persons are required to respond to a 
collection of information unless such collection displays a valid OMB control number. The valid 
OMB control number for this information collection is 1875-New. The time required to 
complete this information collection is estimated to average one hour per response, including the 
time to review instructions, search existing data resources, gather the data needed, and complete 
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and review the information collection. If you have any comments concerning the accuracy of 
the time estimate(s) or suggestions for improving this form, please write to: U.S. 
Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns 
regarding the status of your individual submission of this form, write directly to: Beth 
Franklin, Policy and Program Studies Services, U.S. Department of Education, 400 Maryland 
Avenue, S.W., office 6W219, Washington, D.C. 20202. 


Do you have any questions before we begin? 
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Management and Leadership 


1) How long have you served as your state’s director of RF/Title I/Reading-Literacy director? 


2) What are your RF-related job responsibilities (past and/or current) * 
Competitions for RF funding (selection process, making grant awards) 

Budget allocations at state level, at local level 

Professional development 

Outreach to non-RF schools 

Monitoring RF evaluation data 

Other 

Job responsibilities and approximate percentage of time for RF/Title I vs other 
responsibilities beyond RF/Title I/Reading-Literacy 


3) Your organizational/management and reporting structure 
e Describe the organization chart for Reading First/Title I/Reading/Literacy in the state (to 
whom do you report and who reports to you); describe coordination of state-level reading 
initiatives. 


Reading First Funding at the State Level 


The following questions ask about Reading First funding in your state, both in terms of what has 
already been funded and in terms of planning for future cohorts under a reduced federal budget. 


4) Number and nature of RF awards to date in your state* 

e Average school-level award amount, number of schools funded to date, average duration of 
grant awards, how award amounts are determined e.g., is funding based on student 
enrollment, an equivalent amount for each funded school, other) state minimum/maximum 
level award 


5) In your state, did most districts and schools implement the RF program in all four grades at 
once or did they roll out the program one grade at a time?* (If the latter, could you estimate 
the approximate percentages of districts/schools that rolled out RF implementation?) 


6) Describe any unanticipated exits from participation in Reading First for districts/schools 
within your state (e.g., have any districts and/or schools had funding revoked or voluntarily 
withdrawn from participation in Reading First, has your state added any new cohorts mid- 
cycle?), and whether they have had an impact on subsequent funding decisions.* 


7) Describe your state’s response to the 61% reduction of federal funds for Reading First. 


a) What changes do you anticipate in terms of funding levels and grant duration?* 

b) What are your plans to find other federal and state funding sources for awarded 
districts and schools? 

c) Describe efforts to work with other State-level employees about increasing spending on 
reading (for example, Special Education or Title I coordinators)? 

d) What are your plans for any additional RF cohorts ?* 


Appendix 1: Interview Protocol 39 


Reading First Student Achievement Data 


In this section I will ask you about student achievement data analysis, use, and dissemination. 


8) Whois responsible for preparing data on student performance in RF schools and to whom 
are these data released or distributed? * 


9) Who at the state level uses RF student achievement data and for which purposes? (e.g., 
planning professional development, technical assistance, assessing core and supplemental 
materials, assessing districts and school performance, determining future funding, 
comparing RF and non-RF schools and districts) 


10) What are expected district- or school-level actions as a result of published/released data 
(e.g., changes in professional development, assessment, instruction)? 


11) Describe changes you have observed (since your state began to implement RF) in state, 
district, and school capacity to use RF data. 


Technical Assistance from State Reading First Program 


Next, I will ask you questions about various elements of the RF program both BEFORE the budget cuts 
were implemented and how you anticipate these RF elements will be affected AFTER the budget cuts are 
implemented. 
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12) Describe technical assistance provided by your office to RF district and school staff (e.g., 
nature, frequency, support for, coordination of, participants, changes to). TA includes 
both PD activities and specific help to choose and implement core, supplemental, and 
intervention programs, administering assessments, using data to determine effectiveness of 
programs, etc.* 

a) BEFORE the RF budget cuts 
b) How do you anticipate the TA your office provides will be affected AFTER the RF 
budget cuts are implemented 
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Outreach and Communication with Schools and Districts 


13) Describe your state’s outreach to non-RF districts and schools. (e.g., what information and 

support were non-RF districts and schools asking for and what was provided?)* 

a) BEFORE the budget cuts* 

b) How do you anticipate your state’s outreach to non-RF districts and schools will be 
affected AFTER the budget cuts are implemented* 

c) What challenges have you experienced as a result of your state’s outreach to non-RF 
districts and schools and do you anticipate these will change after the budget cuts are 
implemented?* 


e Prioritization of non-grantee districts and schools for outreach 

e Methods used to inform and engage non-RF schools and districts(with whom do you 
communicate ) 

e = Level of participation of non-RF districts and schools in RF-sponsored conferences, 
professional development, TA, other activities 


14) Describe coordination between RF and other state-level initiatives and programs (e.g., 

higher education, special education, ELL, etc.) 

a) BEFORE the budget cuts 

b) How do you anticipate your relationship with other state-level initiatives and programs 
will be affected AFTER the budget cuts are implemented 

c) What challenges have you experienced in coordination with other state-level initiatives 
and programs and do you anticipate these will change after the budget cuts are 
implemented? 


Diversity of Reading Programs within the State 


15) The Department of Education's RF Program Office is interested in learning how many RF- 
funded districts in each state have reading instruction that is distinctly different across RF- 
and non-RF schools. Can you please tell me if you have the following, and if so, which 
districts: * 

e Districts where the reading program is not the same for RF and non-RF schools 

e Districts where the required assessments differ between RF and non-RF schools 

e Districts where the requirement for a 90-minute reading block differs for RF and non-RF 
schools 

e Districts where reading coaches are differentially available to RF and non-RF schools 
Districts where reading-related professional development opportunities differ for RF and 
non-RF schools 

e Districts where the methods for identifying struggling readers and interventions differ for 
RF and non-RF schools 
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Long Term Sustainability of Reading First 
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16) 


17) 


What evidence have you seen that RF practices are being adopted in non-RF districts and 
schools (e.g., commonalities in use of materials, professional development topics, uses of 
assessments)? 

a) BEFORE the budget cuts were implemented 


b) How do you anticipate these adopted RF practices will be affected AFTER budget cuts 
are implemented? 


Which elements of Reading First you believe will be sustained and why, given the 61% cut 
to the budget for the Reading First Program in fiscal year 2008. 
e = =Which aspects of reading instruction currently practiced in RF schools will remain? 


Thank you for participating in this interview. 
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APPENDIX 2: STATE SUMMARY PROFILE WORKSHEET 


The state summary profile worksheet (see Exhibit A2.1 on next page) was a tool used by interviewers on 
the study team to record relevant information about individual states prior to contacting interviewees; 
such information was obtained from individual state applications for Reading First funding, state Reading 
First websites, federal websites about Reading First, and posted state documents. The purpose was to 
organize information about each state in one document that interviewers could review before conducting 
interviews. Completing this form also reduced the burden on interviewees, as they were asked to confirm 
the accuracy of information about their respective state programs. In addition to the specific information 
to be filled in on the worksheet, interviewers recorded additional notes or comments for each section to 
collect as much background information as possible from public data and records. 
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Exhibit A2.1: State Summary Profile Worksheet 


State Summary Sheet 

State: 

State Award Date: 

State Award Amount: 

Number of RF Districts: 

Number of RF Schools: 

Number of Cohorts of Districts/Schools Funded: 
Additional comments or notes on State Summary: 


Assessments Required 
List Names and Grade Levels of Required Assessments: 


Additional comments or notes on Assessments: 


Resources Available from State to RF Districts and Schools (Posted on state websites) 
Circle yes or no for each item below to indicate availability: 

Professional development opportunities: Yes or No 

Professional development providers: Yes or No 

Materials: Yes or No 

Assessments: Yes or No 

Other resources: Yes or No 

Additional comments or notes on resources available from the state to RF districts and schools: 


Evaluation Data or Reports (Posted on state websites) 
Circle yes or no to indicate whether data or reports are listed or downloadable: Yes or No 
Additional comments or notes on evaluation data or reports available: 


Notes: 

' Data on Reading First awards and number of districts and schools obtained from http://www.sedl.org/readingfirst/report- 
awards.html 

> Data on individual states obtained from each state’s Department of Education or Reading First website. 

Sources: State websites and interviews with RF and/or other state personnel. 
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APPENDIX 3: DESCRIPTION OF INTERVIEW DATA COLLECTION 


Data Collection Procedures 


Senior researchers conducted telephone interviews with state staff in the fall of 2008. Before 
data collection, the researchers participated in a training session to ensure that all interviewers 
shared a common understanding of the study background, objectives, design, and schedule; the 
purpose of the state director interviews; each question in the interview protocol and associated 
probing questions; the logistics of scheduling and conducting interviews; the calendar for data 
collection, analytic meetings and analyses; and data security, data recording, and submission 
procedures. 


Prior to conducting the state director interviews, the study team sent a packet of materials to each 
state’s RF, Title I and reading or literacy director that included an introductory letter signed by 
an ED official, a study fact sheet, and a brief overview of the key questions and topics to be 
discussed in the telephone interview. State personnel received these read-ahead materials so they 
could compile any extant information that could be useful in answering the interview questions. 
Researchers individually scheduled telephone interviews with each state-level respondent. 


Approach to Analysis 


Data Management and Organization 

Each question and sub-question on the interview protocol corresponded to a topic of interest for 
addressing the evaluation question and was assigned a topic code. This preliminary coding 
scheme was programmed into NVivo, a qualitative analysis software program used to annotate 
and organize the information produced from the interviews. Once an interview was complete, 
the interviewer’s notes were transcribed, edited, reviewed, and formatted by topic code. The 
interview data were then imported into the software, which organized them across interviews 
according to these topic codes (e.g., how states used data, outreached to non-funded districts, 
coordinated with other reading initiatives). Once organized, the analysis team reviewed the 
information in each topic code to determine whether the preliminary topic codes were at an 
appropriate level of specificity and to revise them as necessary. 


Data Coding 

In the first phase of analysis, information classified under each topic code was reviewed to assess 
its relevance to that topic code. That is, information collected in response to one question, and 
hence initially coded under that question’s topic code, was reviewed to determine whether it 
would more appropriately be linked to other questions or topic codes, and reclassified as needed. 
The analysis benefited from the inherent flexibility of the software, which accommodated the 
addition of new topic codes and cross-linking or modifying coding schemes while analyses were 
in process. This flexibility allowed for a more in-depth analysis of initial findings, as well as 
provided a mechanism for coding new findings as they emerged. 
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The second phase of analysis required coders to carefully review the content of text categorized 
within a topic code to look for content driven themes. Individual interview responses were 
grouped together according to like answers. Where appropriate, quantitative data were also 
coded. 


To ensure consistency during both phases of coding, all coders received training on the initial 
coding scheme and met during the coding process to discuss questions and interpretations of how 
the data should be coded. Coding discrepancies that arose were resolved via consensus during 
regular team meetings at which staff reviewed the data, identified and refined major themes, and 
revised coding schemes as necessary. 


Data Analysis 

After the analysis team had reviewed the interview data, documents were combined into a single 
master database and parsed by content topic to produce reports. Each report contained all 
information on a given topic and was sorted by interviewee or state characteristics (e.g., director 
type, years of RF funding), allowing for greater analytical freedom and power within a structured 
analysis. The data were characterized by the extent to which themes were raised by different 
groups within each topic. Responses were then summarized qualitatively and quantitatively for 
reporting, and quotes were selected if they helped to illustrate key points or better articulate a 
particular finding. 


46 Appendix 3: Description of Interview Data Collection 


The Department of Education’s mission is to promote student achievement 
and preparation for global competitiveness by fostering educational 
excellence and ensuring equal access. 


www.ed.gov 


